All for One: Multi-Modal, Multi-Tasking

We introduce Attention Teams, a unified framework for training multiple models end-to-end for multiple tasks. This paper explores banding together simple Gated Recurrent Units in teams coordinated by a high-level attention mechanism. The attention mechanism is operated by a GRU leader that determines how relevant the output of each GRU teammate is to the current input. This kind of coordination shows promise for more efficiently using parameters partitioned over small models for multi-tasking across Visual Question Answering and Sentiment Classification tasks. We hope that the idea of Attention Teams will inspire others to try more complex units within the teams, as the attention gives the architecture both the ability to communicate structure as well as the ability to intelligently ensemble units.